Recognising User Intentions in a Virtual Environment

Human-centred design in the adaptive virtual interface is about supporting the intentionality of the user. An operator has a purposeful motivation in undertaking activity in the virtual environment. This can range from taskfocused activity in a vocational sense, such as telepresence surgery or rehearsing mission tactics in a flight simulator, to the purely recreational exploration of a cyberspace. In each case, the adaptive virtual interface needs to monitor the user, and compare user behaviours to models of possible actions. A method is demonstrated for constructing procedures from the spatio-temporal data that describe action plans of agent/entities in a virtual environment. These are required for testing candidate operator intentions against operator action history. This is explored experimentally in the context of flight simulation, and we present a method for learning action plans in real-time. Three components are required: an appropriate ontology (model of operator task performance), an appropriate virtual environment architecture (accessibility of data and image generation databases) and a learning procedure (which relates the data stream to the domain ontology). Relational machine learning methods are applied to traces of pilot behaviour in flight tasks. The simulator is enhanced to provide descriptions of the out-the-window world view in the data trace as well as operator control actions and simulation internal state variables. Control actions are relatable to the achievement of world goal states. Further work has generalised this from user intention modelling, to the modelling of other simulated entities.